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ABSTRACT 
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Preface 



The original edition of Assessing the Difficulty of Reading Materials The 
Noun Frequency Method v^as written by Warwick Elley in 1975. Prior to that 
the 'noun count', as the technique is popularly called, had been described 
in Education. The word list used for identifying common nou 3 was the late 
Gordon Arvidson s NICER Alphabetical Spelling Lists, At that time this was 
the most comprehensive list of New Zealand children's written vocabulary, 
containing words graded by frequency of use. The 1975 publication extended 
the list of nouns to some 2000, further clarified the procedure and included 
a discussion of the rationale and validity of the noun frequency method 

It I?, perhaps not widely appreciated that the origins of the 'noun count* 
tectinique lie in the development of the Progressive Achievement Test of 
Reading Comprehension by Warwick Elley and Neil Reid. One of the initial 
problems faced in developing the content referencing of this test and advanc- 
ing the "level scores' concept, was to find a simple yet reliable method of 
measuiing readability for New Zealand children. The ingenious solution, 
which became the cornerstone of the technique, was to regress the average 
frequency of nouns within each test passage on to the mean ages of a 
sample of children demonstrating adequate comprehension of prose pass- 
ages. Although the noun count' technique originated as part of the test 
development process, it soon became clear that it would develop a life of 
its own. 

For this first revision of Assessing the Difficulty of Reading Materials The 
Noun Frequency Method, Warwick Elley is joined in authorship by Cedric 
Croft. Much of the data on which this revision is based have come from the 
latter's studies of the word-use of primary children, in particular those under- 
pinning the NZCER publication Spell-Write, An Aid to Writing, Spelling and 
Word Study As well as incorporating this new material, the revision contains 
an extended valiaity section and introduces computer versions of the noun 
frequency procedures, which are being developed for IBM-compatible, 
Apple, BBC and Commodore PC's, 

The two authors have combined previously with Colin Cowie to produce 
The New Zealand Basic Word List. Revision of the Dolch Basic Sight Vocab- 
ulary List Their backgrounds m reading, studies of children's writing vocab- 
ularies and educational measurement, make them well suited to continue 
tne development of this very pract *al approach to assessinc the readability 
of materials. 

The 1975 edition of Assessing the Difficulty of Reading Materials: The 
Noun Frequency Method has proved to be a popu!..r and valuable resource 
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for teachers, authors, book publishers and others who may need a quick, 
accurate and reliable measure of readability. I am confident that the Revised 
Edition will also prove popular and continue to be a widely-used professional 
tool. 

The New Zealand Council for Educational Research Is pleased to publish 
this work and the associated computer programs, and looks forward to 
continuing its important statutory function of bringing sound research-based 
materials to leachers at all levels of tht? education system. 

Ian D Livingstone 
Director 
NICER 
June 1989 



i Introduction 



Simple objective methods of estimating the difficulty of reading materials for 
children have been sought for over half a century. In that period over 100 
systems* of measuring readability have been devised. None however, has 
proven to be entirely satisfactory, most are cumbersome to apply and con- 
flicting estimates of difficulty levels are not uncommon. 

Sensitive observers of children and books may in the course of time, ac- 
quire sufficient skill to require no further aids in judging t^e reading difficuity 
of d story. Nevertheless, the majority of teachers, librarians, j-arents, test de 
velopers, and others interested in matching books with chikiren, have felt, 
the need for an independent and objective method for judging readability, 
without having to resort to a major research investigation. 

It cannot be denod that reading difficulty is only one aspect of suitability, 
one which must be considered along with interest level, print size, layout 
ai.d pictures, as v\ ell as the purposes and competencies of the reader. The 
co.Tiplexity of the reading task is however, an important Ingredient to take 
account of when selecting materials for children to read. Many a young po- 
tentially interested reader has been discouraged by a needless succession 
of unfamiliar words and cumplicated constructions. Measures of vocabulary 
requirements and stylistic complexity can then support teachers in their ef- 
forts to entice children toward a love of books. 

In the course of the development of a set of standardised tests of reading 
for general use ir New Zealand schools, Elley (1967) investigated a range 
of existing, widely-used readability techniques. The hope v.as that one such 
method could be identified which would make it possible tc grade reading 
passages according to difficulty level. This would enable teachers to draw 
conclusions from the standardised test results about the level of difficulty of 
materials that each child could oe expected to comprehend adequately. 

The traditional formulae proved disappointing on the whole. Generally 
they were time-consuming and arduous to apply and the results were bv no 
means consistent. Therefore new approaches were investigated. Three 
Imes of thought led to the development of the readability method which is 
described in the following pages. 



Rationale 

Firstly, It has been found in stuotos of verbal learning, that a strong relation- 
ship exisis between the frequency with which a word occurs m writing and 
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its 'meaningfulnebs. According to Underwood and Schuiz (1960) the 
higher the meaningfulness of a verbal unit the more frequently that word has 
been experienced/ Apparently the words we see often are the words we 
rate high in meaningfulness. The psychologists definition of meaningful- 
ness is, of course, more akin to the layman's idea of association value than 
to any philosophical concept of meaning, but it is reasonable to assume that 
the correlation between meaningfulness and difficuky level, as encountered 
by children, is likely to be high. Thus, an index of frequency of exposure to 
certPin verba! units should be helpful. 

Secondly, it was noted that previous studies of readability had shown vo- 
cabulary load to be a consistently good index of reading difficulty. This, de- 
spite the fact that vocabulary measures were very unrefined, classified usu- 
ally on a two-point scale, as 'familiar' or unfamiliar'. Thus Dale and Chall 
(1948) showed a correlation of 0.68 between their criterion of reading diffi- 
culty and the pro^^ortion of words outside the Dale list of 3.000. This was the 
highest of five rer dability indices. Gray and Leary (1935) too. found that the 
number of unfrimiliar words correlated more highly with their cntenon than 
any of 21 nd\ces investigated, and similarly prom.sing results had been 
shown by Spache (1958). Lorge (1944), Forbes (1952). and Bormuih (1966). 
In recent reviews of readability, Harrison (1980) and Dav.son (\^Q5) also 
riade the point that vocabulary is the most important factor in determining 
readability. Since satisfactory results were obtained in these studies with 
only 3 two-point scale of vocabulary load, it was concluded that still better 
results would be produced with a scale which took account of various de- 
gree:> of familiarity of the words of a passage, rated accoiding to their fre- 
quency of occurrence. 

The third line of approach to a new readability method stemmed fronri a 
study of langjage. Students of linguistics point out that m a sentence, cer- 
tain content-bearing words carry most of \\ c meaning of a cor:,munication. 
while others are used primarily to indicate relationships amongst them. 
Nouns, verbs, adjectives and adverbs carry the load of meaning in most 
communications. Relational words, such as prepositions, pronouns, an cles 
and conjunctions, probably set fewer limits < n comprehension, and are usu- 
ally found in similar proportions in simple and complex prose alike. How- 
ever, the rating of such words for readability analyses adds considerably to 
the laboriousness of the exercise, without ar.y apparent improvement in val- 
idity Tiere seemed to be justification then, in limiting the classes of words to 
be rated to the main content-bearing words. 

Preliminary and subjective analyses of complex prose passages, taken 
from comparatively difficult writers such asToynbee, Galbraith, Whitehead, 
Thackeray and Dickens showed that their difficulty tended to roside more in 
the nouns than the verbs or adjectives. Adtled rea£>ons for concentrating on 
nouns came from a 'pos* hoc' analysis of a related expenment m reading 
comprehension. In this investigation, 70 secondary school pupils completed 



a 'Cloze* test often prose passages, each containing approximately 1^0 
words. Each sixth word had been eliminated from the passages, and the 
pupils* task was to fill the blanks with the aid of the context. After marking, 
the results were classified by part of speech iri cder to determine the rela- 
tive difficulty levels for each class of word (soe Table 1). Clearly, Jn$ 
proved to be the most difficult class of words to replace correctly. It seems 
reasonable to conclude from this result, that nouns are surrounded by less 
redundancy than other parts of speech; their oomprehension then, is more 
critical to an understanding of the passage. 

Table 1 : 

Cloze Test Completions Classified by Part of Speech 

Nouns Verbs Prons. Adjs. Advs. Co.iis. Preps. 



Possible 408 611 216 170 394 

No. Correct 



Actual 2ii8 403 296 318 64 101 293 

No. Correct 

% Correct 28.6 72.4 72.6 52.1 38.9 59/ >4 



Some corroboration for this view was found in a study of more than 
10,000 errors in oral reading made by 100 beginning readers. Clay (1966) 
showed that many of these errors are corrected from context by the children 
themselves. Of these self-corrections, 72% occurred equivalent gram- 
matical categories, and amongst these, the nouns e Mown to have the 
lowest rate (2i%), of correct substitution. Figures for other word classes 
were verbs (35%), adjectives (33%). preposition?. (46%), and pronouns 
(60%). Apparently readers who cann-^i attach meaning to the nouns in a 
passage can gain little help from the context. 

Thus, on the basis of these analyses of vocabulary and grammatical 
structure there seemed good reason to conduct empirical studies on the 
noun as a basis for a readability check. Nevertheless, verbs and adjectives 
were also examined along with nouns in the empirical studies descnbed 
below. 



Word Frequency Counts 

To provide a convenient and accurate method of grading the nouns accord- 
ing to famihanty, several word frequency counts were examined the origi- 
nal edition of the present text. The Dale-Chall (1948) lists, commonly used 

9 
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Other studies, were considered unsuitable because they provide only two 
categories - familiar and unfa.Tiillar. The Thorndike-Lcrge (1944) list does 
have a gnded scale, but it is not sufficiently discriminating £.mongst the high 
frequency words to be useful with children's reading matter. The English 
word list of Edwards and Gibbon (1964) is also graded, but contains fewer 
than 1.400 words all told, not enough for the purpose intended. Michael 
West's (1953) General Service List of English Words is a potentially useful 
list m that it classifies the words ac:ording to the frequency of occurrence of 
their various meanings and uses, but it too contains fewer than 2.000 words, 
and would be cumbersome to use. It is apparent too that all the abov*-- lists 
are now beginning to date. More recen:iy. as part of this revision, the com- 
prehensive American Heritage Wo.d Frequency Book by Carroll Davies and 
Richman (1971) was investigated, but was felt to be too doseiy iinKed with 
American school texts to be of major use in New Zealand. 

In fact, as the original analysis was \o show, the most useful lists were 
those derived from childien's own writing, rather than from books wntten by 
adults Amongst these v/ere Radfords (1960) Australian word count, and 
Jhe list prepared for the Board of Education of the City of New York (1954). 
.based, in turn, on Rirrsland's (1945) word count of children s writing. The 
New Zealand adaptation of tNs lis!, prepared by Arvidson (1961) as xhe 
NZCER Alphabetical Spelling List, Book 2, was felt to have the most poten- 
tiai or the original edition of the present text. Its 2.700 words were classified 
into seven levels of frequency, it was convenient to use. and suited to local 
conditions To improve its discrimination at the upper end. an eighth level of 
270 words, compiled by Holder (1965) was added. The list of nouns pub- 
lished in the original edition of this book has now been replaced bv an un- 
dated list. 



Revised List of Nouns 

The rrvised list contained in the Appendix contains 2.050 nouns, graded 
tnto eight levels on the basis of frequency of occurrence in the free writing of 
New Zealand primary school children. The major soiree of data for this revi- 
sion came from a national survey of primary children's wnting by Croft 
(1983a). wh'ch formed the basis of the NZCER publication SpefWnte An 
Aid to Writing, Spelling and Word Study. An account of the major findings of 
this survey is contained in Croft (1983b). and the conditions under which the 
samples of free writifiQ were obtained may be found m Croft {1983c) After 
further analysis of the 198.854 running words from this survey. 182 nouns 
were deleted ffom Elley s original list, mostly from Levels 7 and 8. In addi- 
tion. a number of words from lower levels which are used less often because 
of changes m social conventions, curriculum, currency, measun^ment etc 
were also omitted In making the final judgements on these matters, the au- 
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Ihors were guided by the two NZCER frequency counts listed above, and 
other major word lists,' such as the American Heritage Word Frequency 
Soo/f. Carroll era/.(1972). 

Examples of omissions were: acre (7), arithmetic (4), automobile (5), 
chores (7), elf (8), fiddle (8), igloo (7), multiplication (8), negro (7), noun (6). 
oats (4), ox (7). pail (6), penny (2). pier (7). pint (3). plough (6). pronoun (7), 
quart 13). shepherd (7), sleigh (4), thread (5). ton (8), verb (5). wag (3). wig- 
wam (8), and yarn (8). 

New words Introduced in this revision reflect corresponding changes in 
linguistic usage, greater recognition of Maori language, the advent of com- 
puter technology and other obvious influences. Amongst the additions to the 
new list are: barbecue (5). cancer (4). cassette (8). cheque (8). disco (6). 
display (3), diver (6), gear (3), halloween (7). hui (7). junk (5). kilometre (4). 
laser (7). lump (2). marae (7). metre (4), monster (3). motorbike (8). pakeha 
(7). pitcher (5). skateboard (3), sky rocket (5). stereo (8). supermarket (6). 
tonne (8), tour (6). trampoline (8), trike (5). video (6), and word-processor 
(6). 

Apart from the omissions and additions, approximately 49 per cent of the 
words retained from the original 1975 list have had their frequency levels 
amended. As would be expected, words with altered levels tend to be at or 
above Level 5. and the changes have rarely been more than two levels. 
Examples of exceptions to these two general rules are: president, moved 
from 3 to 6; seam and anchor, from 5 to 8; climate, from 4 to 7; price, from 
6 to 2; weekend, from 8 to 6; and speaker, from 8 to 5. More typical of the 
smaller changes to existing levels are. beach, moved from 3 to 1, dollar, 
from 3 to 2; space from 6 to 4, wharf and flash from 7 to 5; writer and tele- 
scope, from 8 to 6. In the other direction typical examples are: answer, 
moved from 2 to 3, candy, from 3 to 5, museum, from 3 to 4; character and 
member, from 6 to 7. kindness and leather, from 5 to 7, flock and grammar, 
from 7 to 8; and ruler, from 6 to 7. 

In summary then, the mam changes between the 1975 list of nouns and 
the 1989 revision are as follows: 
(i) 184 dated words were removed from the 1975 list, 
(il) 227 new words were listed, 

(Hi) 900 or 49 per cent of the words retained from the 1975 list have been 
placed at a different level. 



II Procedure 

After several approaches had been tried and validatevi. the following proce- 
dure was adopted as the easiest and most accurate .nethod of assessing 
the difficulty level of a story or article. 
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1. Select from the story or sele:ilion to be rated, three* passages long 
enough to contain at least 25 different nouns in each. If the sty''=^ vanes with 
difficulty, it is advisable to choose the more complex passages, since these 
usually set the upper limit on comprehension. Otherwise, select passages 
at random from throughout the story. If a textbook is being rated, and stu- 
dents would bo expected to learn new vocabulary and content in sequential 
fashion, then passages drawn from the first triree chapters would provide 
better estimates than those from the latter half of the book. 

2. Identify each noun in the passages to 'ce rated, and using the list of 
nouns in the Appendix, look up and record the frequency level of each noun. 
Any noun nof appearing in this list of 2,050 is rated Level 9. 

e.g., ball (1), balloon (6), ballot (9). 
Note: 

(a) If a noun occurs mere than cnce in a passage, count it only once. 

(b) Do not count people*'* narjes. 

(c) All other proper noii..s (days, ethnic groups, countries, cities, institu- 
tions, etc.,) follow the same rules as all common nouns. 

e.g., Monday (2), Maori (2), Canada (5), Wellington (9), Beehive (9). 

(d) Give regular plural nouns the same rating as their singular form, 
e.g., bottle(s) (4). 

(e) Hyphenated words follow the normal rules, 
e.g., word (2), word-processor (6), word-bank (9). 

(f ) Any doubtful words are best omitted from the analysis. 

e g , gerunds, abbreviations, acronyms, and recently-coined words. 

3 Calculate the mean frequency level, i.e., add up the frequency level 
numbers and divide by the number of nouns. 

4 Refer to table 2 (page 14) to determine the approximate age group for 
which the material is suitable for instructio.nal purposes. 



'Some research on the reliability of other readability formulae suggests that four or five 
passages will provide a more accurate indication when whole books are to be rated, if the 
first three passages provide similar estimates, then such a precaution would be unneces- 
sary. 
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Worked Example 

Selection from New Zealand School Journal (Part 3, Number 3 1986). 



THE LIMPET MYSTERY 

When you were a small limpet, you found this rock. Between tides, 
you were always in this exact S20l. I watched you grow to fiFlhe 
bumps and hollows till there were no gaps between your shell and the 
rock. I think if there were g?ps, you v/ould dry out and die. This rock 
you are on is not very hard. ^ 'our shell has worn an oval groove in it. 

When the tide is in, you are like a sheep slowly chewing your way 
around the rock, grazing on algae and tiny seaweeds . The fastest you 
move is about one centimetre a minute . I have measured and timed 
you. 

When the tide starts going out. you turn and race for home (limpet 
racing, that is) at nearly three centimetres a minute. BeloreThe rock 
ones out, you must be back in your oval groove. If you dor t make it. 
you may not be able to seal yourself to the rock. You might dry out and 
die. You always make it. 

How do you find your way back if you have no ej^ to see the way? 
I thoughtsihiaybe you were following your scent back. I scrubbed your 
trail with a scrubbing brush. You still got backT" 

I put bricks in your path. You went around them. You went over 
them. I put a cage over your home. After two days, you seemed to 
forget. You stopped trying to get back. I took away the cage and lifted 
you back to your place . 



limpet 

rock 

tides 

spot 

bumps 

hollows 

gaps 

shell 



9 
3 
7 
4 
5 
9 
6 



groove 

sheep 

way 

algae 

seaweeds 

centimetre 

minute 

home 

racing 



eyes 

scent 

trail 

brush 

bricks 

path 

cage 

days 

place 



1 
9 
5 
3 
5 
5 
3 
1 
1 



TOTAL: 122 



Average frequency level of nouns = 122/26 = 4.69. 

This passage would be classified as suitable for average 9V2-IOV2 year old 

readers. 
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Table 2: 

Suitable Age Levels for Readabiilty Ratings Obtained from Noun Counts 
Mean Noun Frequency Rating Approximate Age Range 



Below 2.8 7 to • years 

2 8 to 3.2 yVzioBVz years 

3;2to3.6 8 to 9 years 

3 6 to 4.0 8V2to9V2 years 

4 0 to 4.4 9 to 10 years 
4 4 to 4 8 9V2 to IOV2 years 

4 0 to 5.2 10 to 12 years 

5 2to5.S 11 to 13 years 
5 6 to 6.0 12 to 14 years 
6.0 to 6.4 13 to 16 years 
Over 6.4 15 years and over 



This table was prepared by testing individually, 100 children with the graded 
passages of the Progressive Achievement Tests of Reading Comprehension. 
Pupils who answered at least three out of five questions correctly on each pass- 
age moved on to the next passage until they reached their limit. These li^iiits 
were plotted against the pupils' ages, and trend lines were drawn to produce toe 



111 Evaluation of the Method 

Numerous studies have been conducted to establish the validity of the noun 
frequency method, in five independent but similar investigations, a number 
of prose passages were chosen at random from newspapers, books, 
magazines, school texts and reading tests. The passages ranged in diffi- 
culty from simple material suitable for average 7 or 8 year olds, to complex 
pro<;e liKely to provide some challenge for competent adolescent readers. 
Each selection contained from 120 to 150 words. In the first two investiga- 
tions eight passages were used; in the other three, the number of passages 
was increased to 12, 14 and 16 respectively. The number of 'judges' used in 
the fivt investigations was 10, 12, 43, 50 and 43 respectively. 

Teachers were asked to 'rank the passages in order of difficulty of com- 
orehension, from easiest to hardest'. The average ranks of the teachers pro- 
vided the criterion of difficulty in the first two investigations. In the last three, 
the teachers' rankings were supplemented with the averaged rankings of 



top-stream 12 to 15 year old pupils. As the opinions of the teachers and the 
pupils correlaied over 0.90 in these studies, their rankings wer?3 sub- 
sequently combined. 

Teachers' and pupils' opinions were used as a criterion measure oecause 
they provided a more direct and sensitive measure than the usual melhod of 
graded texts or results from reading comprehension tests. Studies con- 
ducted by Klare (1974) and Harrison (1980) have confirmed that pooled 
teacher judgements tie an excellent criterion for judging the validity of vari- 
ous readability estimates. Furthermore, the combined-opinion criterion Is 
less cumbersome than the 'cloze' test method proposed by Bormuth (1966). 
The latter method was used, however, in a later validity study, reported 
below. 



Table 3: 

Rank Correlations Between Judges' Criterion of Difficulty and Readability 
Estin lates for Five Independe Studies 



Readability 
Estimate 


Series 
1 


Series 
2 


Series 
3 


Series 
4 


Series 
5 


Median 


Noun Frequency Level 


.95 


.91 


.85 


.90 


.88 


.90 


Noun plus Adjective 














Frequency Level 


'>0 


.89 


.82 


.86 


.92 


.89 


Noun plus Verb 














Frequency Level 


.97 


.88 


.64 


.82 


.89 


.88 


Lorge Formula 


.87 


.93 


.80 


.69 


.92 


.87 


Syllables per 100 Words 


.96 


.92 


.76 


.74 


.85 


.85 


Dale-Chall Formula 


.78 


.93 


,68 


.85 


.92 


.85 


Proportion of Abstract 














Nouns 


.70 


.88 


.82 


.83 


."^9 


.82 


Proportion of Unfamiliar 














Words 


.80 


.81 


.71 


.83 


.82 


.81 


Verb Frequency Level 


.72 


.83 


.07 


.49 


.83 


.72 


Adjective Frequency 












.64 


Level 


.64 




.48 


.63 


.89 


Sentence Length 


.76 


.98 


.54 


-.04 


.45 


.54 


Ratio of Prepositional 














Phrases 


.83 


.43 


.10 


.14 


.41 


.41 



The correlations of the best predictors of readability, as found in the five 
investigations, are listed in Table 3. The fact that all these correlations are 
slightly higher than those reported in previous studies is attributable largely 
to the greater rar:ge of diinculty m the passages used. In the third series, this 
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opread of difficulty was reduced, with a corresponding drop in validity coeffi- 
cients. From Table 3, It can be seen that the correlations for the noun fre- 
quency are consistently higher on average than any of the other measures 
examined. Moreover, the noun frequency method contributes substantially 
to the predictive value of he adjective and verb frequency counts, with 
which it was combined for xs^ perimental purposes. 

The Lorge and Oale-Chall formulas, which require- the combining and 
weighting of such variables as sentence length, prepositional phrases, and 
unfamiliar words, are not only less effective in these studies, but are consid- 
erably more labcious in their calculation. Yet in a large-scale survey in the 
United Kingdom, Harrison ,1980) showed that the Dal3-Chall formula was 
the best of several indlcatoro of readability, as measured against the pooled 
judgements of teachers. 

The Fry formula, which is based on tiie number of syllables per 100 words 
and sentence length, is not readily comparable vvith the other estimates 
since it provides a location on a two-axis graph, rather than a single numen- 
cal score. Its dependence on a structural dimension, sentence length, ap- 
pears to make It somewhat less valid, ac were all such methods examined. 

Among the single dimension measures studied, the best were all indirect 
estimates of vocabulary load, but were less refined and less useful as pre- 
dictors of difficulty level th an was the average noun frequency level. 

These vocabulary variables are all highly intercorrelated and were not 
found to increase their correlations with the criterion when combined in vari- 
ous ways. Nor was it found profitable tc combine the noun frequency count 
with any of the structural variables. In the third series, for instance, a combi- 
nation of nun frequency level plus number of words per sentence, produc- 
ed a 'alidity coefficient of 0.68, or 0.17 less than the noun count alone. 
Where discrepancies did occur between the vocabulary load predictors and 
the criterion, they seemed to be attributable more to some unique feature of 
the author's style, or to a specialised use of vocabulary. For instance, the 
correlations for the noun counts in the third study were lowered mainly by 
three passages, a technical description of door-hanging, a biographical 
passage which was heavily packed with detail, and a narrative passage 
from Tolstoy whose style was considered unusual by the judges, although 
the vocabulary was simple. // doubtful whether any objective index could 
predict these intangible qualities without becoming more cumbersome than 
the determination of the criterion itself. 

Another early validity check was made using the cloze test criterion as 
recommended by Bormuth (1966). The prose passages used in this study 
were selected and rated by Bormuth himself, in the following manner. They 
were administered in 'rotating cloze test format' to carefully chosen, repre- 
sentative samples of American school children. In this procedure, each fifth 
word was omitted from the passage. The student's task was to fill the gap 
with the appropriate word, on the basis of his/her understanding of the con- 
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text Five parallel forms of each passage were used, thus allowing each 
word to be tested once. The 'cloze index* for each selection was taken as 
the average percentage of words that were correctly inserted by the five 
groups of pupils tested. This index is still considered by many researchers to 
be the most satisfactory criterion of reading difficulty, despite the cumber- 
some nature of its production. 

Every passage consisted of 100 words, which normally generated about 
20 nouns in each. This is less than the optimum number of the noun count 
procedure but it did produce a useful guide. Passages having fewer than 
18 nouns were omitted from consideration. Highly technical selections were 

also excluded. . ^ ^ ^ 

The 63 passages were classified according to topic, and noun count rat- 
ings were made. These ratings were then correlated with the 'cloze* ratings 
as provided by Bormuth. The results are presented in Table 4. 

The results for these passages again show consistently high validity Fur- 
thermore, the omission of one unusual passage from the Current News 
series would have increased the correlation coefficient from 0 82 to 0 88 

It IS of interest to note that, of over 70 readability indices which Bormuth 
investigated in h^s 1966 article, the highest correlations with ^cloze' difficulty 
were produced by such measures as number of letters per independent 
clause (0.81) and several measures of vocabulary load (between 0 70 and 
0 80) Bormuth s investigations included more passages, however, many of 
which were judged a priori' to be too short or too technical for the noun 
count procedure. 

Table 4: 

Correlations Between Noun Count Ratings and 
Cloze Test Difficulty on Selected Passages 

Subject Number of Passages Rating 

Literature 
Biology 
Current News 
Geography 



0.89 
0.90 
0.82 
0.92 



Other Word Frequency Lists 

In each of the five earlier validity studies reported in Table 3, the NICER Al- 
phabetical Spelling List, Arvidson (1960), was used as the base sou'^ce for 
checking the frequency level of nouns in the oassages. In the fifth of these 
studies other available word counts were also investigated '^xamined first 
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were the lists prepared by Radford 0960), Edwards and Gibbon (1964), and 
the Board of Education of the City of New York (1954). These lists were pre- 
pared on the basis of the written work of Australian, English and American 
children, respectively Wright's (1965) list which is based on the frequency of 
words in common reading materials was also usee. The correlations for 
noun frequency counts, graded accoding to these lists, were all slightly 
lower than those produced with the NZCER list, but were suffic'citly high to 
suggest that they could be used with profit in the countries for which they 
were intended. 

Radford's (i960) list contains nearly 3,000 words, but uses only six levels 
of frequency Its predictive validity of 0.84 in this study compared reasonably 
well with the 0.88 of Arvidson (i960). The Edwards and Gibbon (1964) hst 
contains only 1,347 words, classified (at the seven years over level) into six 
degrees of frequency Nevertheless, when nouns were graded bv this list a 
correlation coefficient of 0.80 with the criterion was obtained. With a more 
extensive list the validity would doubtless improve. 

The Spelling List of the Board of Education of the City of New York con- 
tains 5.000 words classified into ten frequency levels and is, therefore, very 
useful for a sensitive vocabulary count. Its validity coefficient of 0 87 would 
probably be even higher if used in an American context, since it was af- 
fected in this study by thu occurrence of a number of words in the passages 
unfamiliar to New Zealanders, but not to Americans. 

Wright's (1965) word cc jnt, based as it was on predominantly adult read- 
ing matter, rather than children's writing, correlated 0.71 with the critenon 
This result was obtained using the first five levels only (i.e., the most com- 
mon 5,000 words). Nevertheless, the discrepancies which lowered the cor- 
relation would not, in the opinion of the writers, be reouccd by any adapta- 
tion of this or parallel lists. 

Comprehension appears to depend more on familiarity of words used by 
pupiis m their own writing, rather than of words they encounter m their read- 
ing Hence our reliance on data from Croft's (i983a) national survey of pri- 
mary writing, to update the list of nouns contained in the Appendix. 



Recent VfJidity Studies 

A further study conducted by Elley (1982) made use of a different cntenon of 
readability - the average difficulty indices of comprehension questions 
based on reading passages in standardised tests. This cnterion was widely 
used in early research on readability but has the obvious disadvantage that 
the questions can vary in difficulty according to the approach of the test con- 
structor. Nevertheless, as all the tests were standardised, they had been 
Item analysed and revised, and would presumably have t^een screened for 
extreme cases of fluctuation in difficulty 
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Table 5: 

Correlations between Noun Count Ratings and Passage Mean Difficulty Indices 
of Items in Standardised Reading Tests 



Test 



No. of Noun 
Passages Frequency 



Noun + % Words Syllables 
Verb on Dale per 100 
Frequency List words 



Sentence 
Length 



NSW Rdg.A 
NSWRdg.B 



PATRdg.A(Pt3) 
PATRdg.B(Pt8) 
Stanford Rdg.A 



6 
6 
9 
9 
6 



.84 
.75 
.81 
.88 
.84 



.75 
.80 
.67 
.88 
.72 



.82 
.80 
.62 
.60 
.82 



.75 
.79 
.41 
.42 
.15 



.38 
-.32 
.25 
.52 
.30 



Median 



.84 



.75 



.80 



.42 



.30 



Table 5 shows the results for five such tests which have passages of 20+ 
nouns and for which accurate difficulty indices could be obtained. Although 
the correlations are lower for the reasons staieo above, the trends are the 
same as in earlier validity studies. The Noun Frequency Method, on its own, 
is consistently more valid than the other procedures. 

In order to explore further the reasons for the success of this apparently 
sensitive measure of vocabulary load, Elley (1982) also compared the ef- 
fects on pupil comprehension of simplification of difficult prose passages 
usin. three different approaches: viz. reducing sentence complexity, reduc- 
ing vocabulary load, and reducing both simultaneously. Four difficult prose 
passages of 150-200 words were selected from textbooks and encyclo- 
paedias and rewritten. These passages v;ere presented as lexical cloze 
tests (with content words only deleted in the ratio of 1 word i^ 6±1). The 
sample of students was 48 14-15 year olds drawn from two Christchurch 
urban secondary schools. The content of the passages covered history, sci- 
ence, literature and biography. 

The passages were revised to provide three alternative forms of the 
original. 

(i) The simplified syntactic (SS) revision eliminated complex sentences, 
passives, subordinate clauses and circumlocutions, with minimal 
changes in meaning. The mean sentence length was reduced from 32.4 
words ivanced adult level) to 11.3 words (10 year old level). 

(II) The simplified vocabulary load(Sy) revision retained the long complex 
sentences bu! replaced unfamiliar words with familiar ones. The mean 
noun frequency count was reduced accordingly from 6.96 (advanced 
adult) to 4.62 (10 year old leve" 
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(iii) The simplified syntax and vocabulary load (SSV) revision changed both 
dimensions simuIlaiieoLisly, while again preserving the intended mean- 
ing of the authors, and producing combined effects on sentence length 
and vocabulary load comparable lo the other two separate revisions.'' 
Three weeks after the original cloze tests, the same students took a ran- 
dom sample of two of the 16 possible rev.i>t^d forms of each test (4 topics x 4 
modes), again as a lexical cloze test. Students were required to replace con- 
tent words, which were also deleted in the ratio of 1 in 6±1. All 16 passages 
were read by at least 6 students. In varying sequences to counteract prac- 
tice effects. 

Residual gain scores were calculated, from pre-test to post- iest, and a 
three-way analysis of variance performed .o test the effects of syntax, vocab- 
ulary load and reading ability on comprehension gain. As predicted, nnean 
gains for students who read the original versions and -^'mpiified syntax (SS) 
versions were small, while those who read the simplit<ed vocabulary re- 
vision (SV) and the combined revision (SSV) were nearly double those of the 
original passages, with or without syntactic changes. The ANOVA results 
(Table 6) showed highly significant effects for vocabulary (F^ 16.74. p^.001). 
but no significant effects for syntax or ability level (F - 1.44 and 0.02 respec- 
tively) The only interaction effect to reach significance was that of vocabu- 
lary and ability level (F = 8.61, p<.001), which appears to indicate that 
weaker readers were more confused by the difficult vocabulary than were the 
more able readers. 



Table 6: 

Analysis of Variance Showing Effects of Prose 
Simplification on Gaips m Student Comprefiension 



Source 



S, Squares 



df 



Mean Square 



F 



A (Ability) 
B (Syntax) 
C (Vocabulary) 



0.10 
8.76 

102.10 
3.01 
52.50 
6.51 
3.01 

536.92 



0.10 
8.7G 
102.10 
3.01 
52.50 
6.51 
3.01 
6.10 



0.02 
1.44 
16.74" 
0.49 
8.6r' 
1.07 
0.49 



AB 
AC 
BC 
ABC 



Within Cells 



88 



Total 



712.91 



95 



•*p<.001 





A plausible interpretation of these results is that the main difficulty students 
have in comprehending complex prose lies with the vocabulary load, rathe*- 
than the sentence structure. Hence, a sensitive index of vocabulary load is 
required in order to assess the difficulty of prose passages intended to be 
read with understanding. Certainly, this study was confined to high school 
students, but it produced similar effects across all topics, expository or narra- 
tive. Complex syntax may of course present problems for very young child- 
ren, but it is of little consequence for most high school students. 

It IS worth noting that critics of objective readability measures have pointed 
to empirical studies such as those of Davison and Kantor (1982) which show 
no effects of syntax simplification, and use such findings to denigrate the for- 
mulae. The Christchurch study reported above is consistent with such 
studies with respect to syntax, but suggests that the potential value of vocab- 
ulary load is not affected by these conclusions. 

Why then, have measures of syntax been used repeatedly in most reada- 
bility formulae? Possibly because authors have a natural tendency to match 
vocabulary load with .sentence complexity. When they write for children they 
reduce both simultaneously. Hence, correlational studies would show a rela- 
tively high relationship between both variables. It takes an experimental 
study to show whether both are critical for their effects on reading com- 
prehension. 

Other researchers, e.g.. Mason (1986). have criticised the naivety of cur- 
rent readability measures because they ignore indices of abstraction based 
on such linguistic features as the level of nomlnalisation of nouns, the fre- 
quency of words of Latin and Greek origin, and of verbs in the passive tense 

However, an analysis of the graded noun list in the Appendix shows that 
the Noun Frequency Method is already an accurate indicator of abstraction, 
presumably because they are highly correlated with frequency of use by 
children of varying age levels. Thus, the nouns found at Level 1 are concrete 
(baby, back, bah. bat, bath . . .) while most of those found above Level 7 tend 
to be abstract (advantage, advertisement, affair, association, attempt ) 

Other critics downplay the role o: objective validity measures on the 
grounds that they do not identify the -causes of aifficully in prose, and can 
therefore be misleading for those who simplify text for children. The Noun 
Frequency Method does not claim to be any more than an indicator of diffi- 
culty, based on statistical relationships. But why does it appeai to be so sen- 
sitive? A number of psycholinguistic studies, e.g.. Loosen (1981). Wearing 
(1974). Thios (1975). Clark and Card (1969). have shown that the content 
words in a text do carry the load of meaning, and that nouns are more impor- 
tant m this regard than verbs, adjectives or function words. In Loosen (1981) 
for instance, nouns were rated for importance and were recalled better than 
verbs, adjectives, adverbs or function words. Intuitively it makes sense tha* 
the subject and object of a sentence, usually nouns, must be understood 
the text IS to be processed. The bulk of the studies quoted above and ev 
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dence from cloze tests, miscue analysis, recall tests, ratings of importance 
and cues for recall all point in this direction. Hence, the efficiency and value 
of an indicator based on noun familiarity is not very surprising. 

There is clearly a need for more research to refine existing indices of vo- 
cabulary load, but there is sufficient evidence to show the value of the current 
procedure and justify its continued use by teachers, researchers, librarians, 
editors and others, as a practical method of assessing the reariabilityof text. 

Computer Applications 

The increased availabilty of personal computers has made a computensed 
version of the Noun Frequency Method a viable alternative. Earlier versions 
in BASIC had been written for Apple machines and Commodore 64 's. These 
have proved valuable for local use, and although developed with the pub- 
lisher's permission and assistance, were nevertheless subsidiary to the man- 
ual method. 

Ian Livingstone has developed programs for Apple.Commodore.BBC i 
IBM compatible machines, as part of this current revision. These will pro > 
an alternative method of calculating readability levels and will pave the way 
for individuals to build up their handy files of material graded by readability 
level. 

These programs will accept nouns in isolation from the remainder of the 
text provided they are entered individually. In this instance there is no need 
to the nouns. Alternatively, running text may be entered from a word- 
processor, but if this option is used, each noun to be rated must be identified 
before the program will mn. These two methods of entering nouns provide 
a measure of flexibility for users. 

The print out coniing from these programs will show the title of t!ie extract, 
the readability esti** ,ate and the nouns entered or 'flagged', along with their 
frequency levels. This latter feature will be of benefit when there is a require- 
ment to rewrite text to a specified readability level, as> the nouns which need 
to be replaced will be easily identified. The added flexibility introduced by 
the programs bring a new dimension t. ^e Noun Frequency Method, which 
v/e are sure will be fully utilized by increasing numbers of users. 
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IV Conclusions and Cautions 



Assessing k Jabilily is a complex matter. The method descnbed in this text, 
although simple to apply, is net a panacea. II should be seen as an aid. to be 
used sensitively and cautiously, and supplemented by a knowledge of child- 
ren's reading interests, and of any idiosyncracies in the style of the reading 
material which may not be reflecte'j in an objective measure of vcchbulary 
load. Unconventional sentence structure, high density of ideas, uncommon 
techniCo.1 terms, frequent use of idioms, metaphoric language, and simple 
words used in unusual ways, all could distort the estimates of readability ob- 
tained in ^ few cases. An unevenness in difficulty of vocabulary load on the 
part of the author would also complicate assessment, no rratter how it cs 
carried out. In addition, the method should not be used with: 
(i) poetry 

(li) beginning reading material 

(iii) technical articles 

(iv) material having a specialised vocabulary. 

It IS most appropriate for children's prose, desigr <3d for 7 to 16 year old read- 
ers. For textbooks with specialised vocabulary, the cloze procedure, al- 
though somewhat less convenient, nas much more to recommend it. 

Despite these cautions and limitations, there appears to be a proper and 
useful rok for readability indices. Many teachers have found it nelpful in 
selecting bocks and ar*'cles for children. A classified guide to SchoolJournal 
stones has been prepared by the Resources Divistcr! of the Ministry of Edu- 
cation with \hd aid of the Noun Frequency Method. 

In conclusion, it must be emphasised that books do vary tremendously in 
their ease of comprehension. Children likewise vary in their ability to com- 
prehend. To match a child with a suitable book is an important task which re- 
quires undoubted professional skill. Wise adults will know when and how to 
use an objective :ndex to assist them in this task. It is to help in such an en- 
terprise that the Noun Frequency Method of assessing readability was de- 
veloped and published. 



or; 
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Appendix 



Revised List of Nouns Graded by Frequency of Use 



A 

accident 5 
account 6 
ache 8 
act 6 
action 8 
activity 6 
addition 6 
address 5 
adult 8 
advantage 8 
adventure 5 
advertisement 8 
advice 8 
aeroplane 2 
affair 7 
afternoon 2 
age 3 
agreement 7 
agriculture 8 
aid 5 
aim 6 
air 2 
airline 6 
airport 4 
alann 5 
alley 8 
alligator 4 
alphabet 6 
ambulance 7 
America 3 
American 4 
amount 5 
amusement 8 

O 
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anchor 8 
angel 6 
angle 8 
animal 3 
ankle 7 
answer 3 
ant 6 
anybody 4 
anyone 3 
anything 2 
apartment 7 
ape 7 

appearance 8 
apple 1 
appreciation 8 
apricot 7 
April 4 
apron 8 
area 6 
argument 8 
arm 3 
army 3 
arrow 5 
art 4 
artist 7 
assembly 5 
assignment 8 
association 7 
astronaut 7 
atom 6 
attack 6 
attempt 8 
attention 6 
attic 8 
audience 6 



auditorium 8 
August 4 
aunt 2 
auntie 2 
aunty 2 
Australia 4 
Australian 5 
author 6 
autumn 4 
avalanche 5 
avenue 6 
average 7 
axe 7 

B 

baby 1 
back 1 
backstop 8 
bacon 8 
bacteria 7 
badge 6 
bag 3 
bait 5 
baker 8 
baking 3 
balance 8 
ball 

ballet 7 
balloon 6 
banana 4 
band 3 
bandage 7 
bang 2 
bank 3 
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banner 8 
bar 5 
barbecue 5 
bark 5 
barn 4 
barrel 7 
barrier 7 
base 3 
baseball 4 
basement 7 
basin 7 
basket 3 
basketball 5 
bat 1 
bath 1 
bathroom 7 
battery 7 
battle 5 
bay 4 
beach 1 
bead 6 
beak 5 
beam 5 
bean 6 
bear 2 
beard 6 
beast 3 
beauty 6 
bed 1 
bedroom 3 
bedtime 8 
bee 4 
beef 7 
beer 7 
beggar 8 
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beginning 3 


boss 5 


behaviour 7 


bother 7 


being 2 


bottle 4 


bell 3 


bottom 3 


belt 5 


bounce 8 


bench 6 


bound 8 


bend 6 


boundary 8 


berry 5 


bow 5 


bet 4 


bowl 4 


Bible 7 


box 2 


bicycle 4 


boy 1 


bike 1 


boyfriend 7 


bill 4 


bracelet 8 


bin 4 


brain 7 


bird 1 


brake 7 


birth 6 


branch 4 


birthday 2 


brand 8 


biscuit 4 


brass 8 


bit 3 


brat 7 


bite 4 


bread 2 


blackberry 8 


break 3 


blackboard 6 


breakfast 3 


blade 8 


breast 6 


blanket 5 


breath 5 


blast 5 


breeze 8 


blind 6 


brick 5 


block 4 


bridge 3 


blood 3 


bridle 8 


bloom 7 


Britain 5 


blossom 7 


broadcast 8 


blow 4 


brook 7 


board 3 


broom 7 


boat 1 


brother 1 


body 2 


brownie 8 


bomb 6 


bruise 7 


bone 4 


brush 3 


bonfire 8 


bubble 6 


bonnet 8 


bubblegum 6 


book 1 


bucket 5 


booklet 8 


bud 8 


boom 6 


bug 7 


boot 5 


buggy 4 


border 8 


building 3 


bore 8 


bull 5 
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bulldozer 6 


car 1 


bullet 7 


caravan 6 


bump 5 


carDon o 


bunch 6 


card 3 


bundle 8 


cardboard 7 


bunk 6 


cardigan 8 


bunny 2 


care 2 


bus 1 


caretaker 7 


bush 3 


carol 7 


business 4 


carpenter 6 


butcher 8 


carpet 6 


butter 3 


carriage 6 


butterfly 6 


carrot 7 


button i" 


cart 7 




case 4 




cash 8 


c 


cassette 8 


cab 7 


castle 4 


cabbage 5 


Ual 1 


cabin 4 


catch 2 


cabinet 8 


catcher 6 


cafeteria 8 


cattle 4 


cage 3 


cause 4 


cake 2 


cave 3 


calendar 6 


ceiling 7 


calf 4 


ceiery o 


call 2 


ceil o 


camel 6 


cellar 7 


camera 7 


cement 8 


camp 2 


cent 2 


campaign 7 


centimetre 4 


can 1 


centre 4 


Canada 5 


ceniury 0 


canary 6 


cereal 8 


cancer 4 


certificate 8 


candle 5 


chain 7 


candy 5 


chair 3 


cane 6 


ch airlift 7 


cannibal 8 


cnaiK 0 


canoe 5 


cnampion d 


canyon 6 


championship 4 


cap 6 


chance 4 


capital 5 


change 4 


captain 3 


chapter 7 



character 7 


clothes 2 


copper 7 


crumb 7 


charge 6 


clothing 5 


copy 7 


crust 7 


charm 8 


cloud 5 


corn 6 


cry 1 


chart 6 


clown 4 


corner 2 


cub 2 


chase 5 


club 3 


cost 3 


cup 7 


check 6 


clump 6 


costume 6 


cupboard 6 


cheek 8 


coach 7 


cot 3 


cure 8 


cheer 7 


coal 4 


cottage 5 


curl 6 


cheese 4 


coast 5 


cotton 3 


curtain 5 


chemist 8 


coat 2 


couch 6 


cushion 7 


cheque 8 


cobweb 5 


cough 7 


custom 7 


cherry 7 


cocoa 8 


council 8 


customer 8 


chest 5 


coconut 7 


counter 7 


cut 2 


chick 7 


coffee 4 


country 2 


cutting 5 


chicken 3 


coffin 7 


couple 4 




chief 4 


coke 7 


courage 7 


n 


child 1 


cold 1 


course 3 




children 1 


collar 6 


court 4 


dad 1 


chimney 5 


collection 7 


cousin 2 


daddy 1 


chin 8 


college 7 


cover 3 


dairy 5 


China 5 


colour 2 


cow 2 


daisy 6 


Chinese 5 


colt 8 
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wrist 6 
writer 6 
writing 2 



yell 6 
youth 8 

z 

zebra 8 
zero 6 
zone 7 
zoo 4 



X 

x-ray 7 
Xmas 3 



Y 

yacht 6 
yard 6 
year 1 
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